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Executive  summary 

The  central  aim  of  the  project  was  to  develop  computational  models  of  how  individual 
decision-makers  learn  in  real  time  to  anticipate  and  take  into  account  the  risks  and  potential 
consequences  of  their  actions.  The  main  focus  was  on  the  medial  prefrontal  cortex  (mPFC),  an 
area  of  the  brain  known  to  signal  mistakes  as  well  as  the  level  of  difficulty  or  conflict  facing  the 
decision-maker.  The  research  effort  involved  iteratively  developing  computational  models  and 
testing  their  predictions  with  fMRI,  leading  to  further  refinements  of  the  model.  The  original 
goal  of  developing  a  model  of  risk  prediction  was  achieved.  Further  effort  yielded  a  more 
general  model  of  how  both  good  and  bad  potential  consequences  are  learned  and  anticipated. 

The  model  predictions  were  validated  by  numerous  behavioral  and  fMRI  studies,  and  the  effort 
also  yielded  an  exact  recursive  model  of  hyperbolic  temporal  discounting.  The  results  overall 
provide  a  new  and  relatively  simple  computational  model  of  consequence  prediction  that 
accounts  for  and  predicts  a  wide  array  of  empirical  data  and  is  well-grounded  in  the  known 
neurobiologically. 

Results 

The  effort  began  by  using  human  fMRI  to  validate  a  prediction  of  the  error  likelihood 
computational  model  (Brown  and  Braver,  2005),  finding  that  the  medial  prefrontal  cortex 
(mPFC)  signals  the  potential  severity  of  adverse  outcomes  of  an  action,  as  well  as  the  likelihood 
of  adverse  outcomes  (Brown  and  Braver,  2007).  This  suggests  an  account  of  mPFC  as  signaling 
the  risk  associated  with  an  action.  A  second  fMRI  study  showed  that  these  risk  signals  are 
specific  to  the  particular  context  in  which  a  decision  is  made  (Krawitz  et  al.,  in  preparation),  and 
a  modeling  study  further  accounted  for  individual  differences  in  risk-taking  as  reflecting  the 
relative  efficacy  of  learning  signals  due  to  mistakes  (Brown  and  Braver,  2008).  These  studies 
fulfilled  the  original  objectives  of  the  project. 

The  PRO  model.  With  the  original  objectives  met,  the  project  effort  was  expanded  to 
include  several  additional  goals.  First,  new  fMRI  evidence  showed  that  mPFC  signals  the 
potential  rewards  as  well  as  the  potential  risks  of  actions,  and  that  these  predictions  compete  at 
the  neural  level  (Alexander  and  Brown,  2010a).  This  led  to  a  generalization  of  the  error 
likelihood  model,  namely  that  mPFC  not  only  predicts  the  risks  of  an  action  but  more  generally 
predicts  the  probability  of  all  potential  outcomes  of  an  action,  both  desirable  and  undesirable 
(Alexander  and  Brown,  In  Press).  Subsequent  project  effort  was  devoted  to  developing  a  new 
computational  neural  model  of  mPFC  to  instantiate  this  hypothesis,  and  the  model  was  able  to 
capture  a  vast  array  of  data  from  human  behavior,  fMRI,  and  ERP,  as  well  as  monkey  single-unit 
neurophysiology  (Alexander  &  Brown,  In  Prep).  In  essence,  the  model  simulates  how  mPFC 
learns  to  predict  the  probable  outcomes  of  planned  actions,  and  these  predictions  are  then 
compared  against  the  actual  outcomes.  Any  discrepancies  between  the  predicted  and  actual 
outcomes  provide  an  error  signal  that  feeds  back  to  refine  the  outcome  predictions.  This  model 
is  now  called  the  predicted  response  outcome  (PRO)  model. 

In  its  original  form,  the  PRO  model  simulation  was  fairly  complex.  In  the  last  year  of  the 
grant  period,  considerable  effort  was  spent  reworking  the  model  to  distill  it  down  into  its  essence 
of  a  few  equations  that  can  still  capture  the  vast  range  of  data  accounted  for  by  the  original 
simulation  (Alexander  and  Brown,  In  Prep). 

Testing  the  PRO  model  with  fMRI.  Armed  with  the  PRO  model,  the  effort  expanded 
to  simulate  the  model  in  new  experimental  paradigms  and  extract  a  series  of  a  priori  predictions. 
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A  series  of  fMRI  and  behavioral  studies  were  then  designed  to  test  the  model  predictions,  and  all 
fMRI  results  were  consistent  with  the  model  predictions.  It  was  found  that  error-related 
activation  in  mPFC  can  be  reversed  when  error  likelihood  is  high  (Jessup  et  al.,  In  Press), 
following  the  model  prediction  that  mPFC  signals  a  discrepancy  of  actual  vs.  expected  outcomes, 
as  distinct  from  a  comparison  of  actual  vs.  intended  outcomes.  It  was  found  that  apparent 
response  conflict-related  activation  in  mPFC  persists  even  when  the  task  conditions  are  changed 
so  that  multiple  responses  are  no  longer  in  conflict  with  each  other  (Brown,  2009),  consistent 
with  the  model  account  of  greater  activity  due  to  predicting  a  greater  number  of  impending 
action  outcomes.  Another  study  now  shows  that  mPFC  is  sensitive  to  the  timing  as  well  as  the 
valence  of  predicted  outcomes  (Forster  and  Brown,  In  Prep).  Yet  another  study  shows  that 
mPFC  has  distinct  functional  subregions  corresponding  with  specific  model  components  (Nee, 
Kastner,  and  Brown,  In  Prep).  In  the  course  of  testing  model  predictions,  it  was  necessary  to 
further  develop  methods  of  quantitatively  fitting  and  testing  model  predictions  directly  with 
fMRI  data.  This  led  to  another  methods-based  paper  on  hierarchical  Bayesian  methods  of  model 
selection  criteria  with  respect  to  fMRI  data  (Ahn  et  al.,  revised). 

Recursive  hyperbolic  discounting  model.  In  the  course  of  developing  models  of 
outcome  prediction,  the  effort  ran  into  a  persistent  issue  in  reinforcement  learning  theory.  On  the 
one  hand,  human  and  animal  studies  of  intertemporal  choice  consistently  show  hyperbolic 
temporal  discounting.  On  the  other  hand,  existing  recursive  models  and  temporal  difference 
learning  models  generally  show  exponential  discounting.  The  effort  was  therefore  further 
expanded  to  develop  the  first  exact  recursive  model  of  hyperbolic  temporal  discounting,  which 
now  allows  simple  and  accurate  online  simulations  of  animal  and  human  choice  behavior 
(Alexander  and  Brown,  2010b). 

Overall,  the  effort  achieved  and  went  well  beyond  the  original  objectives.  Some  of  the 
newer  results  are  still  in  various  stages  of  preparation  and  review,  but  the  results  lay  a  foundation 
of  a  neurobiologically  grounded  mathematical  and  computational  theory  of  how  individuals 
predict  and  take  into  account  the  potential  outcomes  of  their  decisions. 

Personnel  supported 

Joshua  W.  Brown,  Ph.D.  -  PI 
William  Alexander,  Ph.D.  -  Post-Doc 
Adam  Krawitz,  Ph.D.  -  Post-Doc 
Derek  Nee,  Ph.D.  -  Post-Doc 
Woo-Young  Ahn  -  Graduate  student 
Rena  Fukunaga  -  Graduate  student 
Elizabeth  Dinh  -  Research  Assistant 
Sarah  Forster  -  Graduate  student 
Rich  Lewis  -  Research  Assistant 

Publications  resulting  from  the  grant: 

1.  Forster  SE,  Brown  JW  (in  preparation)  Medial  prefrontal  cortex  learns  to  predict  the  timing 
of  action  outcomes. 

2.  Alexander  WH,  Brown  JW  (in  preparation)  Think  before 
as  a  predictor  of  action  consequences 
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3.  Krawitz  A,  Braver  TS,  Barch  DM,  Brown  JW  (in  preparation)  Impaired  Error-Likelihood 
Prediction  and  Evaluation  in  Anterior  Cingulate  Cortex  in  Schizophrenia. 

4.  Nec  DE,  Kastner  S,  Brown  JW  (submitted)  Functional  heterogeneity  of  conflict,  error,  and 
task  switching  effects  within  medial  prefrontal  cortex. 

5.  Ahn  WY,  Krawitz  A,  Busemeyer  JR,  Kim  W,  Brown  JW  (revised)  Neural  Correlates  of 
Subjective  Outcome  Evaluation  in  an  Experience-Based  Decision-Making  Task.  J.  Neurosci. 
Psychol.  Econ. 

6.  Alexander  WH,  Brown  JW  (revised)  Computational  neuroscience  models:  Error 
monitoring,  conflict  resolution,  and  decision  making.  In  V.  Cutsuridis,  D.  Polani,  A. 

Hussain,  T.  Tishby,  andJ.  Taylor  (eds.)  Perception-reason-action  cycle:  Models,  algorithms 
and  systems.  New  York:  Springer 

7.  Krawitz  A,  Fukunaga  R,  Brown  JW  (revised)  Anterior  insula  activity  predicts  the  influence 
of  gain  framed  messages  on  risky  decision-making.  Cogn.  Aff.  Behav.  Neurosci. 

8.  Alexander  WH,  Brown  JW  (In  Press)  Computational  models  of  performance  monitoring  and 
cognitive  control.  TopiCS 

9.  Jessup  RK,  Busemeyer  JR,  Brown  JW  (In  Press)  Error  effects  in  anterior  cingulate  cortex 
reverse  when  error  likelihood  is  high.  J.  Neurosci. 

10.  Alexander  WH,  Brown  JW  (In  Press)  Hyperbolically  discounted  temporal  difference 
learning.  Neural  Computation 

1 1 .  Alexander  WH,  Brown  JW  (2010)  Competition  between  learned  reward  and  error  outcome 
predictions  in  anterior  cingulate  cortex.  Neuroimage  49:3210-3218. 

doi:  10.101 6/j  .neuroimage. 2009. 1 1 .065 

12.  Brown  JW  (2009)  Conflict  effects  without  conflict  in  medial  prefrontal  cortex:  multiple 
response  effects  and  context  specific  representations.  Neuroimage  47:334-341 

13.  Brown  JW,  Braver  TS  (2009)  Executive  function  and  higher-order  cognition: 

Computational  models.  InL.  Squire  {eA.)  Encyclopedia  of  Neuroscience  A\9'i-9'i.  Oxford: 
Academic  Press. 

14.  Brown  JW  (2009)  Multiple  cognitive  control  effects  of  error  likelihood  and  conflict. 
Psychological  Research.  73:744-750.  DOI  I0.I007/s00426-008-0I98-7 

15.  Brown  JW,  Braver  TS  (2008)  A  computational  model  of  risk,  conflict,  and  individual 
difference  effects  in  the  anterior  cingulate  cortex.  Brain  Research.  1202:99-108. 

doi:  1 0. 1 0 1 6/j  .brainres.2007.06.080 

16.  Brown  JW,  Braver  TS  (2007)  Risk  Prediction  and  Aversion  by  Anterior  Cingulate  Cortex. 
Cogn.  Aff.  Behav.  Neurosci.  7(4):266-277 
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Technology  assists: 

Feb-Mar  2009:  On  request,  the  PI  shared  some  raw  countermanding  task  data  from  Brown  & 
Braver  (2005)  with  Glenn  Gunzelmarm  and  Rick  Moore  at  AFRL,  who  are  extending  it  to  studies 
and  models  of  sleep  deprivation  effects  on  cognitive  control  functions. 

Other  interactions  and  presentations  during  the  grant  period: 

1.  Brown  JW,  Nee  DE,  Kastner  S  (2009)  Medial  prefrontal  cortex  shows  a  regional  gradient  of 
monitoring  and  control  functions  consistent  with  a  role  in  outcome  prediction.  Program  No. 
93.2.  2009  Neuroscience  Meeting  Planner.  Washington,  DC:  Society  for  Neuroscience. 
Online. 

2.  Forster  S,  Brown  JW  (2009)  Violations  of  temporal  expectancy  activate  discrete  regions  of 
medial  prefrontal  cortex  in  risk-averse  and  risk-seeking  individuals.  Program  No.  93.1. 

2009  Neuroscience  Meeting  Planner.  Washington,  DC:  Society  for  Neuroscience.  Online. 

3.  Fukunaga  R,  Brown  JW  (2009)  Informative  messages  against  risky  behavior  show 
differential  risk-aversion  related  activity  in  substance-dependent  compared  to  healthy 
individuals.  Program  No.  93.3.  2009  Neuroscience  Meeting  Plarmer.  Washington,  DC: 
Society  for  Neuroscience.  Online. 

4.  Brown  JW  (2009)  Computational  neural  models  of  risk.  Invited  talk.  AFOSR  Joint 
Program  Review  -  Cognition  and  Decision  Program  and  Human-System  Interface  Program. 
Jan  28-30,  2009,  Arlington,  VA 

5.  Alexander  WH,  Brown  JW  (2008)  A  computational  neural  model  of  learned  response- 
outcome  predictions  by  anterior  cingulate  cortex.  Program  No.  682.21.  2008  Neuroscience 
Meeting  Planner.  Washington,  DC:  Society  for  Neuroscience.  Online. 

6.  Brown  JW,  Finn  PR  (2008)  Error  likelihood  and  error  consequence  prediction  effects  are 
inverted  in  the  anterior  cingulate  cortex  of  substance  abusers.  Program  No.  682.19.  2008 
Neuroscience  Meeting  Plarmer.  Washington,  DC:  Society  for  Neuroscience.  Online. 

7.  Ahn  WY,  Krawitz  A,  Kim  W,  Busemeyer  JR,  Brown  JW  (2008)  Disentangling  neural 
processing  of  the  Iowa  gambling  task:  a  model-based  fMRI  study.  Program  No.  681.7.  2008 
Neuroscience  Meeting  Plarmer.  Washington,  DC:  Society  for  Neuroscience.  Online. 

8.  Krawitz  A,  Braver  TS,  Barch  DM,  Brown  JW  (2008)  The  influence  of  working  memory  on 
error-likelihood  prediction  in  the  anterior  cingulate  cortex  and  its  disturbance  in 
schizophrenia.  Program  No.  288.3.  2008  Neuroscience  Meeting  Planner.  Washington,  DC: 
Society  for  Neuroscience.  Online. 

9.  Ashourvan  A,  Brown  JW,  Port  NL  (2008)  A  dynamical  systems  neural  network  model  of 
predictive  remapping  of  the  tilt  aftereffect  preceding  saccadic  eye  movements.  Program  No. 
167.6.  2008  Neuroscience  Meeting  Planner.  Washington,  DC:  Society  for  Neuroscience. 
Online. 

10.  Brown  JW  (2008)  The  basic  science  behind  neuromarketing.  Invited  talk  to  Shoppability 
conference,  Kelley  Executive  Partners,  Kelley  School  of  Business,  Indiana  University. 
October  2008. 
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11.  Brown  JW  (2008)  Risk  predictions  and  cognitive  control  of  decision-making.  Invited  talk 
at  Johns  Hopkins.  October  2008. 

12.  Potter  RF,  Lang  A,  Brown  JW,  Fukunaga  R,  Krawitz  A  (2008)  Brain  activation  during  risk: 
The  influence  of  trait  motivation  on  ACC  activation  during  choice  and  consequence. 
Contributed  paper  at  International  Communication  Association  annual  meeting,  Montreal, 
Canada,  May  23,  2008 

13.  Alexander  W,  Brown  JW  (2008)  Error  likelihood  effects  in  anterior  cingulate  cortex 
modulated  by  average  reward  and  reinforcement  learning.  Contributed  poster  at  Indiana 
Neuroimaging  Symposium,  lUPUI,  April  2008 

14.  Ahn  WY,  Krawitz  A,  Busemeyer  JR,  Brown  JW  (2008)  Neural  correlates  of  decision¬ 
making  processes  in  the  Iowa  gambling  task;  a  model-based  fMRI  study.  Contributed  poster 
at  Indiana  Neuroimaging  Symposium,  lUPUI,  April  2008 

15.  Brown  JW  (2008)  fMRI  methods  -  a  brief  overview.  Invited  talk.  Indiana  Neuroimaging 
Symposium.  lUPUI  campus,  April  2008. 

16.  Fukunaga  R,  Krawitz  A,  Brown  JW  (2008)  Persuasive  messages  against  risky  behavior 
increase  risk  aversion-related  activity  in  the  anterior  cingulate  cortex  and  insula.  Poster 
presentation.  Cognitive  Neuroscience  Society.  San  Francisco,  CA 

17.  Brown  JW  (2008)  Conflict  effects  without  conflict  in  medial  prefrontal  cortex.  Poster 
presentation.  Cognitive  Neuroscience  Society.  San  Francisco,  CA 

18.  Alexander  W,  Brown  JW  (2008)  Error  likelihood  effects  in  anterior  cingulate  cortex 
modulated  by  average  reward  and  reinforcement  learning.  Poster  presentation.  Cognitive 
Neuroscience  Society.  San  Francisco,  CA 

19.  Krawitz  A,  Braver  TS,  Brown  JW  (2008)  The  influence  of  working  memory  on  error- 
likelihood  prediction  in  the  anterior  cingulate  cortex.  Poster  presentation.  Cognitive 
Neuroscience  Society.  San  Francisco,  CA 

20.  Brown  JW  (2008)  Individual  differences  in  medial  prefrontal  cortex,  conflict,  error 
likelihood  prediction,  and  risk  aversion.  Invited  Symposium,  International  Congress  of 
Psychology.  Berlin,  Germany 

21.  Brown  JW  (2008)  Computational  neural  models  of  risk.  Invited  talk.  AFOSR  Joint 
Program  Review  -  Cognition  and  Decision  Program  and  Human-System  Interface  Program 
Jan  22-24,  2008,  Arlington,  VA 

22.  Brown  JW,  Braver  TS  (2007)  Individual  differences  in  medial  prefrontal  cortex,  conflict, 
error  likelihood  prediction,  and  risk  aversion.  Program  No.  232.9.  2007  Abstract 
Viewer/Itinerary  Planner.  Washington  DC:  Society  for  Neuroscience. 

23.  Brown  JW  (2007)  Error  and  conflict,  prediction,  and  the  adaptive  regulation  of  control. 
Invited  talk  at  Conflicts  as  Signals  workshop,  Berlin,  Germany 

24.  Brown  JW  (2007)  The  Role  of  Medial  Prefrontal  Cortex  in  Learned  Risk  Prediction  and 
Aversion.  Invited  talk  at  Max-Planck  Institute  for  Adaptive  Behavior  and  Cognition,  Berlin, 
Germany 
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